Search for: All records

The objective of this study was to evaluate the performance of an XGBoost model trained with behavioral, physiological, performance, environmental, and cow feature data for classifying cow health status (HS). The model predicted HS based on physical activity, resting, reticulo-rumen temperature, rumination and eating behavior, milk yield, conductivity and components, temperature and humidity index, parity, calving features, and stocking density. Daily at 5 a.m., the model generated a HS prediction [0 = no health disorder (HD); 1 = health disorder]. At 7 a.m., technicians blind to the prediction conducted clinical exams on cows from 3 to 11 DIM to classify cows (n = 625) as affected (HD = 1) or not (HD = 0) by metritis, mastitis, ketosis, indigestion, displaced abomasum, and pneumonia. Using each day a cow presented clinical signs of HD as a positive case (i.e., HD = 1), metrics of performance (%; 95% CI) were: sensitivity (Se) = 57 [52, 62], specificity = 81 [80, 82]; positive predictive value (PPV) = 20 [18, 22], negative predictive value = 96 [95, 96], accuracy = 79 [78, 80], balanced accuracy = 69 [66, 72], F-1 Score = 29 [26, 32]. Sensitivity was also evaluated using fixed time intervals around clinical diagnosis of disease as a positive case (Table 1). Our findings suggest that the ability of an XGBoost algorithm trained on diverse sensor and nonsensor data to identify cows with HD was moderate when only days when cows presented clinical signs of disease were considered a positive case. Sensitivity and PPV can be improved substantially when all days within fixed intervals before and after clinical diagnosis are used as positive cases. Table 1 (Abstr. 2614). Sensitivity and PPV for an XGBoost algorithm trained to predict cow health status using fixed intervals before and after clinical diagnosis as positive cases Day relative to CD Se (%) 95% CI PPV (%) 95% CI −5 to 0 58 49, 67 21 16, 25 −3 to 0 55 46, 64 19 15, 24 −5 to 1 69 61, 78 24 20, 29 −5 to 3 81 73, 88 28 23, 33 −5 to 5 86 80, 92 30 25, 34 −3 to 1 67 58, 75 23 18, 27 −3 to 3 78 70, 86 27 22, 31 −3 to 5 83 76, 90 28 24, 33 0 to 3 75 68, 83 24 20, 29 0 to 5 81 73, 88 26 21, 31 −1 to 0 54 44, 63 18 14, 22 0 to 1 63 54, 72 20 16, 25 −1 to 1 66 57, 75 21 17, 26 

Linear temporal logic (LTL) and ω-regular objectives—a su- perset of LTL—have seen recent use as a way to express non-Markovian objectives in reinforcement learning. We in- troduce a model-based probably approximately correct (PAC) learning algorithm for ω-regular objectives in Markov deci- sion processes (MDPs). As part of the development of our algorithm, we introduce the ε-recurrence time: a measure of the speed at which a policy converges to the satisfaction of the ω-regular objective in the limit. We prove that our algo- rithm only requires a polynomial number of samples in the relevant parameters, and perform experiments which confirm our theory. 

Regular decision processes (RDPs) are a subclass of non- Markovian decision processes where the transition and reward functions are guarded by some regular property of the past (a lookback). While RDPs enable intuitive and succinct rep- resentation of non-Markovian decision processes, their ex- pressive power coincides with finite-state Markov decision processes (MDPs). We introduce omega-regular decision pro- cesses (ODPs) where the non-Markovian aspect of the transi- tion and reward functions are extended to an ω-regular looka- head over the system evolution. Semantically, these looka- heads can be considered as promises made by the decision maker or the learning agent about her future behavior. In par- ticular, we assume that if the promised lookaheads are not fulfilled, then the decision maker receives a payoff of ⊥ (the least desirable payoff), overriding any rewards collected by the decision maker. We enable optimization and learning for ODPs under the discounted-reward objective by reducing them to lexicographic optimization and learning over finite MDPs. We present experimental results demonstrating the effectiveness of the proposed reduction. 

We present a modular approach to reinforcement learning (RL) in environments consisting of simpler components evolving in parallel. A monolithic view of such modular environments may be prohibitively large to learn, or may require unrealizable communication between the components in the form of a centralized controller. Our proposed approach is based on the assume-guarantee paradigm where the optimal control for the individual components is synthesized in isolation by making assumptions about the behaviors of neighboring components, and providing guarantees about their own behavior. We express these assume-guarantee contracts as regular languages and provide automatic translations to scalar rewards to be used in RL. By combining local probabilities of satisfaction for each component, we provide a lower bound on the probability of sat- isfaction of the complete system. By solving a Markov game for each component, RL can produce a controller for each component that maximizes this lower bound. The controller utilizes the information it receives through communication, observations, and any knowledge of a coarse model of other agents. We experimentally demonstrate the efficiency of the proposed approach on a variety of case studies. 

Aerial imagery is a powerful tool when it comes to analyzing temporal changes in ecosystems and extracting valuable information from the observed scene. It allows us to identify and assess various elements such as objects, structures, textures, waterways, and shadows. To extract meaningful information, multispectral cameras capture data across different wavelength bands of the electromagnetic spectrum. In this study, the collected multispectral aerial images were subjected to principal component analysis (PCA) to identify independent and uncorrelated components or features that extend beyond the visible spectrum captured in standard RGB images. The results demonstrate that these principal components contain unique characteristics specific to certain wavebands, enabling effective object identification and image segmentation. 

Reinforcement learning (RL) is a powerful approach for training agents to perform tasks, but designing an appropriate re- ward mechanism is critical to its success. However, in many cases, the complexity of the learning objectives goes beyond the capabili- ties of the Markovian assumption, necessitating a more sophisticated reward mechanism. Reward machines and ω-regular languages are two formalisms used to express non-Markovian rewards for quantita- tive and qualitative objectives, respectively. This paper introduces ω- regular reward machines, which integrate reward machines with ω- regular languages to enable an expressive and effective reward mech- anism for RL. We present a model-free RL algorithm to compute ε-optimal strategies against ω-regular reward machines and evaluate the effectiveness of the proposed algorithm through experiments. 

The difficulty of manually specifying reward functions has led to an interest in using linear temporal logic (LTL) to express objec- tives for reinforcement learning (RL). However, LTL has the downside that it is sensitive to small perturbations in the transition probabilities, which prevents probably approximately correct (PAC) learning without additional assumptions. Time discounting provides a way of removing this sensitivity, while retaining the high expressivity of the logic. We study the use of discounted LTL for policy synthesis in Markov decision processes with unknown transition probabilities, and show how to reduce discounted LTL to discounted-sum reward via a reward machine when all discount factors are identical. 

We study the problem of finding optimal strategies in Markov decision processes with lexicographic ω-regular objectives, which are ordered collections of ordinary ω-regular objectives. The goal is to compute strategies that maximise the probability of satisfaction of the first 𝜔-regular objective; subject to that, the strategy should also maximise the probability of satisfaction of the second ω-regular objective; then the third and so forth. For instance, one may want to guarantee critical requirements first, functional ones second and only then focus on the non-functional ones. We show how to harness the classic off-the-shelf model-free reinforcement learning techniques to solve this problem and evaluate their performance on four case studies.